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SUMMARY 


In  1965—1966,  the  Service  Bureau  Corporation,  Houston,  Texas,  programmed  the  IAPG  (Item 
Analysis  Program,  General)  methodology  that  had  been  defined  at  the  6570th  Personnel  Research 
Laboratory,  Lackland  Air  Force  Base,  Texas.  IAPG  consists  of  seven  item  analysis  computer 
programs  that  use  the  responses  to  items  for  which  the  correctness  or  incorrectness  of  a  particular 
alternative  is  not  the  same  for  all  respondents.  IAPG  has  undergone  many  changes  since  its 
inception.  This  report  brings  the  user  up-to-date  concerning  the  card  and/or  file  input  and  printed 
and/or  file  output  capabilities  of  I A  PC. 

The  IAPG  statisticalAnathematical  methodology  is  supported  by  the  computational  formulas 
shown  in  Appendix  A.  The  description  of  the  control  and  data  cards,  input/output  file  layouts, 
samples  of  the  printed  output  and  diagnostic  messages  are  shown  in  Appendixes  B  to  E,  respectively. 
IAPG  is  a  versatile  program:  however,  it  can  be  time  consuming  as  shown  in  Appendix  F.  As  the 
number  of  rases  and/or  items  becomes  large,  the  computer  time  required  could  become  a  limiting 
factor. 
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IAPC:  AN  ITEM  ANALYSIS  PROGRAM  FOR  QUESTIONNAIRE-TYPE 

TEST  INSTRUMENTS 


L  miHODllCTON 


hem  analysis  programs  are  commonly  used  to  determine  the  psychometric  characteristics  of  test 
items  in  order  to  develop  valid  measuring  instruments.  In  the  case  of  aptitude  or  ability  type  tests, 
each  item  has  a  correct  alternative;  however,  in  biographical,  interest,  or  opinion  type 
questionnaires,  no  correct  response  is  designated.  Item  Analysis  Program,  General  (I  APG),  which  is 
a  series  of  seven  item  analysis  computer  programs  referred  to  in  this  report  as  IAPG  1  to  7,  was 
developed  specifically  for  use  with  measurement  devices  containing  items  for  which  the  correctness 
or  incorrectness  of  an  alternative  is  not  the  same  for  all  respondents. 

In  I%5,  the  Service  Bureau  Corporation  was  tasked  under  government  contract  to  develop  a 
computer  program  conforming  to  the  IAPG  methodology  which  was  originally  formulated  at  the  Air 
Force  Human  Resources  Laboratory  (AFHRL).  In  1966,  follow-on  research  was  completed  by  the 
Service  Bureau  Corporation  and  the  IAPG  programs  were  implemented  on  the  IBM  7040  computer 
system  at  AFHRL  (Service  Bureau  Corporation,  1966). 

Since  that  time,  IAPG  has  undergone  several  modifications.  The  purpose  of  this  report  is  to 
acquaint  the  potential  user  with  the  capabilties  of  IAPG  which  has  been  updated  for  use  on  the 
I'NIVAC  1108  computer  system.  Technical  details  are  discussed  that  enable  the  user  to  take 
complete  advantage  of  the  analytical  capabilities  of  IAPG.  This  information  includes  computational 
formulas,  control  and  data  card  descriptions,  file  layouts,  printed  output  samples,  diagnostic 
messages,  and  run  time  examples. 

The  IAPG  computer  programs  are  constructed  so  that  IAPG  I  to  4  and  IAPC  5  to  7  can  be  run 
without  interruption;  however,  the  user  may  run  any  subset  of  either  group  of  programs  as  long  as 
the  input  requirements  for  each  program  are  satisfied.  The  data  set  of  responses,  which  is  normally 
divided  into  three  subsamples,  can  contain  a  maximum  of  five  criteria. 

The  maximum  number  of  alternatives  allowed  per  item  is  six.  with  values  ranging  from  one  to 
six  inclusive.  A  response  for  a  k-alternative  item,  where  the  value  of  k  may  vary  from  item  to  item,  is 
a  set  of  k  elements  where  a  value  of  plus  one  is  assigned  to  the  selected  alternative  and  a  value  of  aero 
is  assigned  to  every  other  alternative.  No  more  than  one  alternative  can  be  selected  for  each  item.  If 
the  number  of  alternatives  for  each  item  is  less  than  six,  an  above-range  response  (the  alternative 
selected  has  a  value  greater  than  six)  and/or  omit  response  (no  alternative  was  selected)  ran  be 
considered  as  an  additional  alternative. 

In  the  following  sections,  each  of  the  programs  is  discussed  in  detail.  The  appendixes  provide 
information  concerning  the  various  statistical  compulations  and  computer  program  specifics.  The 
I  APG  statistiral/malhematiral  methodology  is  supported  by  the  computational  formulas  provided  in 
Appendix  A.  The  description  of  the  control  and  data  cards,  input/output  file  layouts,  samples  of  the 
printed  output  and  diagnostic  messages  are  shown  in  Appendixes  B  to  E.  respectively.  IAPG  is  a 
versatile  program:  however,  it  ran  be  time  consuming  as  shown  in  Appendix  F. 


a  METHODOLOGY  AND  ASSOCIATED  CSPI  TAJITPI  T 


IAPG  1 


IAPG  1  accepts  data  in  the  form  specified  in  Appendix  B  and  produces  the  following  results  for 
each  subsample: 

1. Data  Information  Roster  —  This  is  printed  output  as  shown  in  Appendix  D  containing  the 
total  number  of  cases,  number  of  cases  not  eliminated,  number  of  eliminated  cases,  number  of  cases 
with  no  omitted  responses,  number  of  rases  with  at  least  one  omitted  response,  total  number  of 
omitted  responses  for  all  cases,  and  total  number  of  undefined  criteria  for  all  cases.  (If  a  case  is 
deleted,  an  error  message  is  printed  giving  the  reason  for  the  deletion.  These  error  messages  are 
shown  in  Appendix  E.) 

2.  Response  Proportions  for  Item  Alternatives  Roster  —  This  is  printed  output  for  each  item  as 
shown  in  Appendix  D  and  contains  the  item  identification  number,  proportion  of  r  m  it  ted  responses, 
proportion  of  responses  not  omitted,  and  proportion  of  cases  responding  to  each  alternative.  (The 
analyst  can  examine  this  table  to  locate  items  that  have  been  omitted  too  frequently  or  that  have 
alternatives  associated  with  very*'hitfh  and/or  low  selection  frequencies.  An  item  can  be  deleted  in 
IAPG  2.) 

3. Case  Omit  Information  Roster  —  This  is  printed  output  as  shown  in  Appendix  D  and  contains 
the  case  identification  number,  number  of  omitted  items  for  each  case,  and  identification  numbers  of 
omitted  items.  (The  analyst  can  examine  this  table  to  locate  cases  with  large  numbers  of  omissions.) 

4. Preliminary  Response  Data  File  (PR  PF)  —  This  is  a  file  containing  the  information  shown  in 
Appendix  C.  (The  word  ‘Tile*’  refers  to  either  a  magnetic  tape  or  a  UNIVAC  1 108  FASTR  AN  mass 
storage  file.) 


IAPG  2 


IAPG  2  accepts  the  item  elimination  and  case  omission  information  specified  in  Appendix  B 
and  the  Preliminary  Response  Data  File  developed  in  IAPG  1  as  input.  It  produces  the  following 
results  for  each  subsample.  (If  the  option  to  consider  above-range  and/or  omitted  responses  as  valid 
response  alternatives  is  selected,  the  set  of  response  alternatives  is  augmented  by  one:  however,  the 
augmented  set  must  contain  less  than  seven  response  alternatives.) 

1.  Item  Summary  Information  Roster  —  This  is  printed  output  as  shown  in  Appendix  I)  and 
contains  the  number  of  cases  not  eliminated,  number  of  items  not  eliminated,  total  number  of 
omitted  responses  for  all  cases,  number  of  criteria,  criterion  identification  number,  criterion  mean, 
criterion  standard  deviation,  number  of  criterion  values,  and  the  correlations  between  the 
alternatives  of  each  item.  The  mean,  standard  deviation,  point-biserial  and  biserial  correlation 
coefficients,  and  .01  and  .05  significance  keys  are  printed  for  each  item  alternative.  (The  term  “.01 
and  .05  significance  keys"  is  defined  in  Appendix  A.) 

2.  Item  Summary  Information  File  (ISIF)  —  This  is  a  file  containing  the  information  shown  in 
Appendix  C. 

3.  Final  Response  Data  File  (FRDF)  —  This  is  a  file  containing  the  information  shown  in 
Appendix  C. 


IAPG  3 


IAPG  3  accepts  the  control  card  information  specified  in  Appendix  B  and  the  Item  Summary 
Information  File  developed  in  IAPG  2  as  input  and  produces  the  following  results  for  each 
subsample. 

1.  Item  Key  File  (IKF)  —  This  is  a  file  containing  the  information  shown  in  Appendix  C. 

2.  Roster  of  Significance  Keys  and  Validities  —  This  is  printed  output  for  each  criterion/ 
significance  level  (.01  and  .03)  combination  as  shown  in  Appendix  D.  It  contains  the  item 
identification  number  (including  identification  numbers  for  dummy  items),  sequential  item  count, 
significance  key  for  each  item  alternative,  item  validity,  number  of  items  (excluding  dummy  items) 
containing  at  least  one  nonzero  alternative  significance  key,  and  number  of  dummy  items. 

For  each  item  significance  key  (which  is  the  composite  of  the  item  alternative  significance  keys) 
containing  only  two  of  the  three  possible  values  (+  1,  -1,  0),  dummy  items  are  created  in  the 
following  manner:  (a)  if  the  item  significance  key  is  comprised  of  -t-  I  and  0  values,  the  dummy  item 
significance  key  is  identical  except  that  >1  is  substituted  for  each  0;  (b)  if  the  item  significance  key  is 
comprised  of  -1  and  0  values,  the  dummy  item  significance  key  is  identical  except  that  +  1  is 
substituted  for  each  0  and  (c)  if  the  item  significance  key  is  comprised  of  +  I  and  -1  values,  the 
dummy  item  significance  key  is  identical  except  that  0  is  substituted  for  each  -1.  The  dummy  item 
identification  number  is  the  original  item  identification  number  suffixed  by  the  letter  “A.”  A 
dummy  item  has  the  same  item  validity  as  the  original  item.  Each  dummy  item  that  is  formed  yields 
an  additional  item  key  that  may  be  used  in  the  item  composite  buildup  in  IAPG  6. 

3.  Roster  of  Pattern  Keys  and  Validities  —  This  is  printed  output  for  each  criterion  as  shown  in 
Appendix  I).  It  contains  the  item  identification  number,  keying  patterns  yielding  the  five  highest 
item  validities,  item  validities  corresponding  to  those  keying  patterns,  and  number  of  dummy  items. 

a.  Each  element  of  a  keying  pattern  must  assume  one  of  the  following  three  values:  +  1.  -l.or  0. 

b.  Dummy  item  keys  are  created  only  for  the  keying  pattern  yielding  the  highest  item  validity. 

c.  If  the  item  validity  for  a  keying  pattern  is  negative,  the  signs  of  all  of  the  elements  of  the 
keving  pattern  are  reversed  to  yield  a  positive  item  validity  of  the  same  magnitude. 

d.  A  method  for  generating  unique  keying  patterns  is  given  by  Bottenberg  and  Ghristal  (1964): 
therefore,  item  validities  do  not  have  to  be  calculated  for  all  of  the  3^  ways  in  which  a  k-alternative 
item  can  be  keyed. 

1.  Roster  of  Least  Squares  Weights  and  Validities  —  That  is  printed  output  for  each  criterion  as 
shown  in  Appendix  D.  It  contains  the  item  identification  number,  multiple  correlation  coefficient 
(item  validity),  significance  of  the  item  validity  at  the  .03  level,  least  squares  weight  for  each  item 
alternative,  and  number  of  items  with  significant  item  validities.  (Any  combination  of  the  three 
types  of  keying  options  may  lie  run.) 


IAPG  4 


The  input  to  IAPG  1  is  (he  control  card  information  specified  in  Appendix  B  and  the  Item 
Siimmarv  Information  File  and  Item  Key  File  developed  in  IAPG  2  and  I  VPG  3.  respectively. 


1APG  4  calculates  item  validities  by  using  the  item  keys  from  one  subsample  and  the  item 
alternative  standard  deviations,  item  alternative  point-biserial  correlation  coefficients  and 
correlations  between  the  alternatives  of  each  item  from  a  different  subsample.  The  Item  Key  File 
and  Item  Summary  Information  Fite  may  contain  information  on  two  different  criteria.  Dummy 
items  are  not  used  in  the  cross-validation  calculations  because  the  resulting  information  would  be  the 
same  as  for  the  original  items.  Any  item  existing  in  one  but  net  in  the  other  subsample  involved  in 
the  cross  validation  is  not  considered.  The  Roster  of  Unmatching  Items  is  printed  output  which 
contains  the  items  that  are  not  defined  in  both  the  Item  Key  File  and  the  Item  Summary 
Information  File  for  each  pair  of  subsamples  involved  in  a  cross  validation. 

For  three  subsamples.  IAPG  4  produces  a  Roster  of  Item  Keys.  Validities,  and  Cross  Validities 
as  shown  in  Appendix  D  for  each  of  the  following  subsample  combinations:  (a)  the  Item  Key  File  for 
subsample  1  and  the  Item  Summary  Information  Files  for  subsamples  2  and  3.  (b)  the  Item  Key  File 
for  subsample  2  and  the  Item  Summary  Information  Files  for  subsamples  1  and  3,  and  (c)  the  Item 
Key  File  for  subsample  3  and  the  Item  Summary  information  Files  for  subsamples  1  and  2: 
therefore,  six  rosters  are  printed  for  each  criterion/keying  option  combination.  Each  Roster  of  Item 
Keys.  Validities,  and  Cross  Validities  contains  the  identification  number,  cross  validity,  validity, 
difference  between  the  cross  validity  and  validity,  and  key  for  each  item. 


IAPG  5 


The  input  to  IAPG  5  is  the  control  card  information  specified  in  Appendix  B  and  the  Final 
Response  Data  File  and  Item  Key  File  developed  in  IAPG  2  and  3,  respectively.  For  three 
subsamples  and  a  single  criterionAteying  option  combination,  IAPG  5  produces  the  Roster  of  Item 
Changes/Deletions  and  Keyed  Item  Response  File  Counts  and  a  Keyed  Item  Response  File  (K1RF) 
for  each  of  the  following  subsample  combinations:  (a)  the  Item  Key  File  for  subsample  1  and  the 
Final  Response  Data  Files  for  subsamples  I.  2,  and  3,  (b)  the  Item  Key  File  for  subsample  2  and  the 
Final  Response  Data  Files  for  subsamples  1,2.  and  3,  and  (c)  the  Item  Key  File  for  subsample  3  and 
the  Final  Response  Data  Files  for  subsamples  1 . 2.  and  3.  As  an  option  on  the  M  ain  Control  Card,  the 
user  may  request  that  Keyed  Item  Response  Files  be  produced  from  only  the  Item  Key  Files  and 
Final  Response  Data  Files  of  interest. 

The  Roster  of  Item  Changes/Deletions  and  Keyed  Item  Response  File  Counts  is  printed  output 
as  shown  in  Appendix  D  containing  the  Keyed  Item  Response  File  identification  number,  the 
subsample  identification  numbers  for  the  Item  Key  File  and  Final  Response  Data  File,  the  reason  a 
particular  subset  of  items  was  eliminated,  and  the  identification  numbers  and  total  number  of  items 
in  the  associated  subset.  For  each  item  key  change  specified  by  the  user,  it  also  lists  the  item 
identification  number  and  new  item  key. 

1.  An  item  is  eliminated  in  IAPG  3  if  it  has  an  all  zero  key.  or  if  it  is  not  defined  in  both  the  Final 
Response  Data  File  and  the  Item  Key  File  for  a  particular  subsample  combination. 

2.  The  options  available  to  the  user  are  changing  item  keys  (applies  only  to  original  items)  and 
eliminating  any  subset  of  the  original  items.  For  each  item  that  is  eliminated  or  has  its  key  changed, 
the  associated  dummy  item  will  be  deleted. 

The  Keyed  Item  Response  File  is  a  file  containing  the  information  shown  in  Appendix  ('..  The 
maximum  direct  access  file  size,  which  is  used  in  IAPG  (>.  is  printed.  If  IAPG  5  to  7  arc  run 
consecutively  without  interruption,  this  value  is  automatically  passed  to  I A  PC  (>:  however,  if  I A  PG 
3  and  6  afe  run  separately,  the  value  must  be  present  on  the  Main  Control  Card. 


t 


1APG  6 


The  input  to  IAPG  6  is  the  control  card  information  specified  in  Appendix  B  and  the  Keyed 
Item  Response  Files  developed  in  IAPG  5.  IAPG  6  produces  unit  weighted  item  composites  by 
selecting  items  that  yield  the  largest  increase,  or  minimum  decrease  if  none  of  the  available  items 
yield  an  increase,  in  composite  validity  for  each  iteration  of  the  composite  buildup.  After  each 
iteration,  one  additional  item  with  a  weight  of  plus  or  minus  one  is  included  in  the  composite: 
however,  the  user  mav  select  an  option  on  the  Main  Control  Card  that  will  allow  only  positive  unit 
weights  to  be  used  in  the  composite  buildup.  The  item  having  the  largest  item  validity  is  selected  on 
the  first  iteration.  Items  are  not  available  for  selection  if  they  have  negative  item  validities.  If  an 
original  item  becomes  an  element  of  the  composite,  then  the  dummy  item  associated  with  it  is  not 
eligible  to  beeome  an  element  of  the  same  composite:  likewise,  if  a  dummy  item  becomes  an  element 
of  the  composite,  then  the  original  item  associated  with  it  is  not  eligible  to  become  an  element  of  the 
same  composite.  When  an  item  becomes  an  element  of  the  composite,  it  is  permanently  removed 
from  the  pool  of  available  items  for  that  composite.  IAPG  6  produces  the  following  results: 

1.  Item  Seleetion  Sequence  Roster  —  This  is  printed  output  as  shown  in  Appendix  I)  containing 
the  iteration  number,  identification  number,  and  sign  (blank  indicates  plus)  of  the  item  selected, 
composite  validity,  mean,  and  standard  deviation,  item  validity,  criterion  mean  and  standard 
deviation,  number  of  iterations,  iteration  number  corresponding  to  the  largest  composite  validity, 
number  of  items  defined  in  the  Keyed  Item  Response  File,  number  of  original  items  with  negative 
validity,  number  of  dummy  items  with  positive  validity,  number  of  dummy  items  with  negative 
validity,  and  number  of  items  that  may  be  used  in  the  generation  of  the  composite.  If  the  composite 
validity  decreases  for  at  least  one  iteration,  then  the  printed  output  also  contains  the  iteration 
number  corresponding  to  the  first  decrease  in  composite  validity  and  the  number  of  iterations  in 
which  the  composite  validity  decreased. 

2.  Item  Selection  Sequence  File  —  This  is  a  file  containing  the  information  shown  in  Appendix 

<:. 


IAPG  7 


The  input  to  IAPG  7  is  the  control  card  information  specified  in  Appendix  B.  the  Keyed  Item 
Response  File  (generated  from  the  subsample  i  Item  Key  File  and  the  subsample  j  Final  Response 
Data  File)  developed  in  IAPG  5.  and  the  Item  Selection  Sequence  File  (generated  by  the  Keyed 
Item  Response  File  for  the  subsample  i  item  Key  File  and  the  subsample  k  Final  Response  Data 
File)  developed  in  I  \PG  (».  IAPG  7  generates  an  item  composite  for  the  Keyed  Item  Response  File: 
however,  the  item  composite  must  be  identical  to  the  one  in  the  Item  Selection  Sequence  File. 

For  three  subsamples.  I  \PG  7  can  generate  two  item  composites  (one  for  each  Keyed  Item 
Response  File  generated  from  the  Item  Key  File  for  subsample  i  and  the  Final  Response  Data  File 
for  subsample  j  =k)  for  each  item  composite  defined  in  an  Item  Selection  Sequence  File.  IAPG  7 
produces  the  Rosier  of  Item  Selection  Gross  V  alidation  as  shown  in  Appendix  I)  which  contains  the 
sequential  item  count,  identification  number  and  sign  (blank  indicates  plus)  of  the  item  added,  item 
validilv  from  the  Item  Selection  Sequence  File,  item  validity  from  the  Keyed  Item  Response  File, 
number  of  cases  in  the  Keved  Item  Response  File,  the  criterion  mean  and  standard  deviation,  and 
the  v  alidily  of  the  composite  produced  in  I  At’G  (>.  It  also  contains  the  mean,  standard  deviation,  and 
validilv  of  the  composite  produced  in  IAPG  7. 


<) 
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APPENDIX  ^.COMPUTATIONAL  FORMULAS 


IAPG  1 


Definitions  of  symbols 

N  =  total  number  of  cases  (individuals) 

Xjjk  =  the  response  of  the  k*^1  individual  to  the  alternative  of  the  itR  item  (response  — 
1  if  alternative  selected,  0  otherwise) 


Formulas 

PCRij  = 


the  proportion  of  cases  responding  to  the  j*^1  alternative  of  the  itR  item 


N 

£  X 
k  =1 

“n 


ijk 


IAPG  2 

Definitions  of  symbols 

Nc  =  number  of  cases  with  values  for  the  c1^1  criterion 
Yck  =  the  clR  criterion  value  for  the  k1^1  individual 

Njj  =  the  number  of  cases  with  values  for  the  c1^1  criterion  responding  to  the  jtR 

alternative  of  the  i1^1  item 


"c 

1  xijk 

k  =1  1 


Xijk  = 


the  response  of  the  k*^1  individual  to  the  j1^1  alternative  of  the  itR  item 


Fomtulas 


Xij  = 


the  mean  of  the  jtR  alternative  of  item  i 


standard  deviation  of  the  alternative  of  item  i 


*  Xij  (1  -  Xjj) 
the  cth  criterion  mean 


standard  deviation  of  the  c*“  criterion 


T  v  2 

Yck 

t  =1 

- Yc2 


the  correlation  between  alternatives  a  and  b  of  item  i 


Lia  *ib 


(l-xj(l-x8h) 


the  point-biserial  correlation  between  criterion  c  and  the  j1*1  alternative  of  item  i 


k£=i  XijkYck-NcXijYc 

"cSDcSDjj 

PB"C  is  significant  at  the  .05  (or  .01)  level  if 

PBjjc  VTy2~ 

V  1-PB  ••  2 

v  ijc 


5%  (or  l%)level  of  the  distribution  of  I  with 
Nf-2  degrees  of  freedom. 


The  value  of  the  .05  (or  .01)  significance  key  is:  +  I  if  the  point-biserial  coefficient  of  correlation  is 
positive  and  significant  at  the  .05  (or  .01)  level.  -1  if  the  point-biserial  coefficient  of  correlation  is 
negative  and  significant  at  the  .05  (or  .01 )  level,  and  0  if  the  point-biserial  coefficient  of  correlation  is 
not  significant. 

Bjjr  =  the  biserial  correlation  between  criterion  c  and  the  j*^1  alternative  of  item  i 


then  W 


T 


Z 


PBijc  SDjj 


Where  Z  =  the  ordinate  of  the  unit  normal  distribution  curve  at  the  point  of 
division  between  segments  containing  p  and  q  OTj:  and  1  -X.  ”)  proportions  of  the 
cases.  Z  may  be  computed  (Hastings,  1955)  as  follows: 


if  X;j  ^  .5  set  Q  =Xjj  and  SWT  =1 
if  X”  >  .5  set  Q  =1-X"  and  SWT  =  -l 
J  loge  (1/Q2) 


SWT 


2,515517  +  .802853 W  +  .01 0328 W  2 
1  +  1.432788W  +  .189269W  2  +  .001308W3 


) 


e-(l/2)T 
V"2  7T 


‘  ±±:L 

fctl 


IAPG  3 


Definitions  of  symbols 


Wtij  = 

the  key  (weight)  of  the  j1^1  alternative  of  the  i1*1  item 

PBijc= 

the  point-biserial  correlation  between  the  alternative  of  the  i'*1  item 

criterion  c 

and 

SDij  = 

the  standard  deviation  of  the  j1^1  alternative  of  the  i1^1  item 

NALTj  = 

number  of  alternatives  for  the  i1^*  item 

irab  = 

the  correlation  between  alternatives  a  and  b  of  item  i 

Nij  = 

the  number  of  cases  with  values  for  the  c’^1  criterion  responding  to  the 
alternative  of  the  i*^1  item 

j‘h 

Formulas 

Significance  and  pattern  keys 

rjc  *  item  validity,  correlation  between  criterion  c  and  item  i 

SUMCCV; 


J  CSDSQj 
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Sl'MCCV; 


I 

j=» 


1 


^ij^ijc^ij 


NALT; 

SD”2 

CSDSQ; 

I 

j=« 

lWlij  1 

NALTj-l 
+  2  £ 

NALT; 

£ 

i^.b  Wti«  W*ib  SI>ia  SDib 

a  =  1 

b-a  +  I 

Least  squares  weights 

WEIGHT-  =  the  weight  for  the  j1^1  alternative  of  the  i1^1  item 


k=l 


XijkYck 


item  validity,  multiple  correlation  between  criterion  c  and  the  alternatives  of 
item  i 


ric 's  significant  at  the  .05  level  if 


rir2  (Nr-N ALT;) 
(NALTjHl-r;  2) 


{5%  level  of  the  distribution  of  F  with  NALT;  and  Nc* 
NALT;  degrees  of  freedom. 


IAPG  4 


Formulas 

r.  =  cross  validity,  the  correlation  between  item  i  and  criterion  c.  The  formula  for  r-r 

is  identical  to  the  formula  in  IAPC  3  for  the  item  validity  resulting  from  the  use 
of  significance  and  pattern  keys. 

IAPC  5 


Definitions  of  symbols 

N  =  the  number  of  cases  with  valid  values  for  the  criterion 

c 

Yck  =  the  value  of  the  e,h  criterion  for  the  klh  individual 

Y  =  the  mean  of  the  c1*1  criterion 

S-.  =  the  score  (weighted  response)  of  the  i1^1  item  for  the  k*^1  individual.  It  is  the 

element  of  the  item  key  corresponding  to  the  alternative  that  was  selected  as  the 
response  to  the  item. 

th 

SDr  =  standard  deviation  of  the  c,n  criterion 

Formulas 

Sj  =  the  mean  of  the  scores  of  the  i*^1  item 


c 

I 

k=l 


Sik 


N 


c 


SI) j  =  the  standard  deviation  of  item  i 


r> 


the  correlation  between  criterion  c  and  item  i 


*  SikYck  NcSiYe 


NCSDiSDC 


IAPG  6 


Definitions  of  symbols 


Nc  = 


Wk  = 


the  number  of  cases  with  values  for  the  c1^1  criterion 

unit  weight  used  when  the  k1*1  item  enters  the  composite  (Note:  W  is  +  1  or  -1; 
therefore,  Wk2  =1) 


the  mean  of  item  k 


Sk„  =  the  score  of  the  kln  ite»,:  for  the  n1^1  individual 

SDk  =  the  standard  deviation  of  item  k 

L  =  the  number  of  items  in  the  composite 


Formulas 


AMEAN 


correlation  between  item  k  and  criterion  c 


the  mean  of  a  composite  containing  L  items 


*  *kSk 


the  standard  deviation  of  a  composite  containing  L  items 


L  L-l  I, 

ESD.2  +2  Z  Z 

*  k= 1  m  =k+  I 


*  Skn  Smn 


-Vm/  WkW, 


fi 


composite  validity,  the  correlation  between  the  criterion  and  a 
composite  containing  L  items 


L 

I  WkrkcSDk 
k=l 


UPC  7 

Definitions  of  symbols 

L  =  number  of  items  in  the  composite 

Wk  =  the  unit  weight  (+  1  or  -1)  used  to  add  the  It1*1  item  to  the  composite. 

Skn  =  the  n1^  individual’s  score  for  the  k1^1  item. 

Yc  =  the  mean  of  the  criterion 

SDC  =  the  standard  deviation  of  the  c1*1  criterion 

Nc  =  the  number  of  cases  with  values  for  the  c1*1  criterion 

AMEAN  =  the  mean  of  a  composite  containing  L  items 

SICMA  =  the  standard  deviation  of  a  composite  containing  L  items 


CVALID 


the  composite  validity,  the  correlation  between  the  criterion  and  a 
composite  containing  L  items 


APPENDIX  B:  DESCRIPTION  OF  CONTROL  AND  DATA  CARDS 


Package  1  (IAPC  1  to  4)  Cards:  There  are  eight  types  of  control/data  cards  involved  in  lAPG  1 
to  4.  Five  are  required  and  three  are  optional.  The  order  in  which  they  are  described  is  the  order  in 
which  they  m  u$t  appear  if  needed. 

NOTE  1:  Blanks  and  zeroes  are  interchangeable  unless  stated  otherwise. 

NOTE  2:  All  numerical  entries  should  be  right  justified  unless  stated  otherwise. 

Main  Control  Caid,  required 

cc  Description  Program  Used 


1-6  Card  Identification  must  be  “CONTRL” 

7  1  if  program  1  to  be  run,  blank  otherwise 

8  2  if  program  2  to  be  run,  blank  otherwise 

9  3  if  program  3  to  be  run,  blank  otherwise 

10  4  if  program  4  to  be  run,  blank  otherwise 

1 1  Number  of  subsamples  (if  blank,  assumes  3) 

12  0  if  omits  and  above-range  invalid  (above-range  IAPC  1 

recoded  to  omit  and  case  eliminated  in  IAPC  2) 

1  if  omits  valid,  above-range  invalid  (case 
eliminated  immediately  after  detection  in  IAPG  1) 

2  if  omits  and  above-range  valid  (above-range  recoded  to  omit) 


13  l  if  user  desires  to  eliminate  any  items,  blank  otherwise  IAPC  2 

14  1  if  .01  and  .05  keys  wanted,  blank  otherwise  IAPC  3 

15  2  if  pattern  keys  wanted,  blank  otherwise  IAPC  3 

16  3  if  least  squares  weights  wanted,  blank  otherwise  IAPC  3 

17-21  Criteria  wanted  (in  sequence,  left  justified  in  IAPG  3 

field),  blank  otherwise  (assumes  1 .2.3. 4,5) 

23  I  if  .01  keys  to  be  cross  validated,  blank  otherwise  IAPC  4 

24  2  if  pattern  keys  to  be  cross  validated,  blank  otherwise  IAPC  4 

25  3  if  least  squares  weights  to  be  cross  validated,  blank  otherwise  IAPC  4 

26  4  if  .05  keys  to  be  cross  validated,  blank  otherwise  IAPG  4 

27  An  entry  in  this  column  indicates  that  criterion  1  IAPC  4 

of  the  ISIF  is  to  be  cross  validated  with  an  IKF  criterion. 

The  value  of  the  entry  is  the  ID  of  the  IKF  criterion. 

28  Similar  to  column  27,  but  criterion  2  IAPC  4 

29  Similar  to  column  27.  but  criterion  3  IAPC  4 

30  Similar  to  column  27,  but  criterion  4  IAPC  4 

31  Similar  to  column  27,  but  criterion  5  IAPC  4 


32-35  “NOGO"  option,  if  the  letters  “NOGO”  are  here,  the  program  will 
analyze  the  control  cards  and  print  a  description  of  the  parameters  and 
operations  to  be  performed.  If  IAPG  1  is  included  in  the  “NOGO"  test, 
control  cards  for  each  subsample  will  be  scanned.  Data  cards  must  not  be 
present  when  using  this  option. 

36  1  if  checkpoint  requested  every  hour  (wall  clock),  blank  otherwise 
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Tide  Csrdfs),  required 


The  first  title  card  will  appear  at  the  top  of  every  page  of  output.  As  many  cards  as  desired  may 
be  used  for  title  purposes.  Subsequent  title  cards  will  appear  only  on  the  first  page  of  output.  Card 
column  1  must  contain  standard  FORTRAN  control  characters  (printer  line  control)  and  columns  2 
to  79  will  be  printed  exactly  as  punched. 


End  Tide  Card,  required 

A  card  containing  “END  TITLE”  starting  in  col  2  must  follow  the  title  card(s). 


Data  Control  Caids,  required  for  each  subsample 


cc 


Description 


1-6  “CNTRL1”  (card  identification) 

10-11  Card  number  (01,  02,  etc.) 

For  Card  01  (Subsample  Parameters) 

12  Input  unit  for  data,  5  =cards,  3  *tsp*  or  mass  storage  (FORTRAN  or  COBOL  formatted 
files).  If  data  are  on  a  COBOL  file,  see  columns  29  to  32. 

13-16  Number  of  items  per  case  (The  input  to  IAPC  1  can  consist  of  a  maximum  of  950  items 

per  case  —  see  Appendix  F  for  computer  run  time  considerations) 

17  Number  of  criteria 

18-23  Maximum  number  of  cases  allowed  to  be  eliminated.  Run  terminates  if  exceeded. 
24-28  Identification  numbers  of  criteria  to  be  used  (left  justified  in  field) 

29-32  Block  size  if  COBOL  file  (max  *2500).  LRL  of  file  must  be  14,  i.e.,  card  images. 

For  Cards  02  and  on  (Maximum  Response  Values) 

12-72  Maximum  responses  for  items  in  the  sequence  in  which  they  occur.  (61  one-digit  fields). 

Number  of  cards  needed  determined  by  number  of  items  per  case.  Card  02  contains 
maximum  responses  for  items  1  through  61  and  card  03  starts  with  item  62. 


Data  Caids,  optional,  data  could  be  on  tape 

If  the  data  is  on  tape,  it  will  be  as  card  images  with  the  same  layout  as  the  data  cards.  If  cards  are 
requested,  they  will  be  placed  immediately  following  the  appropriate  data  control  cards. 

NOTE:  The  input  to  IAPG  1  can  consist  of  a  maximum  of  9,999  cases  per  subsample.  See 

Appendix  F  for  computer  run  time  considerations. 
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cc 


Description 


1-9  Case  identification 

10-11  Card  number 

For  Card  01  (Criterion  Card) 

12-23  Value  of  criterion  1  in  F12.8  format  (must  have  either  explicit  decimal  point  or 
understood  decimal  point  between  the  4th  and  5th  positions  from  the  left  of  the  field) 
24-35  Value  of  criterion  2 

36-47  V  alue  of  criterion  3 

48-59  Value  of  criterion  4 

60-71  Value  of  criterion  5 

NOTE:  An  omitted  criterion  must  be  indicated  by  a  blank. 

For  Cards  02  and  on  (Response  Card) 

12-72  Item  responses  in  sequence,  61  per  card 

NOTE  1: 

An  omitted  response  is  indicated  by  a  blank. 

NOTE  2:  End  of  subsample  is  indicated  by  a  data  card  with  9s  filling  the  case  ID  field.  This 

card  is  required. 

hem  Elimination  Cards,  optional,  used  in  1APC  2  to  eliminate  items 

The  cards  contain  identification  numb.;*  of  items  to  be  eliminated  (right  justified  in  three- 
character  fields,  24  items  per  card).  Read  stops  upon  encountering  a  blank  field.  If  the  number  of 
items  to  be  eliminated  is  a  multiple  of  24,  there  must  be  a  blank  card  following  to  stop  the  read. 

NOTE:  The  item  elimination  cards  for  subsample  1  are  first,  subsample  2,  second  and  subsample 

3,  third.  If  items  are  not  to  be  eliminated  from  a  subsample  and  the  item  elimination  code 
on  the  Main  Control  Card  is  1,  then  a  blank  card  must  be  supplied  for  that  subsample. 

Package  2  (IAPC  5  to  7)  cards:  This  package  has  three  required  types  of  cards  and  two  optional 
types.  They  are  described  in  the  order  that  they  are  used. 

Main  Control  Card,  required 

cc  Description 


1-6  “CONTRL"  (card  ID) 

37-40  "NOGO”  only  for  control  card  test 

41  5  if  1APG  5  to  be  run,  blank  otherwise 

42  6  if  IAPG  6  to  be  run.  blank  otherwise 

43  7  if  IAPG  7  to  be  run,  blank  otherwise 

44  1  if  .01  keys  to  be  used 

2  if  pattern  keys  to  be  used 

3  if  least  squares  weights  to  be  used 
5  if  .05  keys  to  be  used 
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NOTE :  Serious  consideration  should  be  given  before  using  least  squares  weights  in  I APG  5 

to  7  due  to  the  possibility  of  multiple-file,  multiple-reel  problems. 

45  Criterion  ID 

46  1  if  user  will  eliminate  items,  blank  otherwise 

47  1  if  user  will  change  keys,  blank  otherwise 

48  Stop  option 

If  blank,  assumes  option  1 

1  causes  stop  after  item  pool  exhausted  or  200  items  are  in  the  composite 

2  causes  stop  after  X  (col  51  to  53)  items  are  in  the  composite 

3  causes  stop  on  Xth  (col  51  to  53)  iteration  after  the  first  decrease  in  the  composite 
validity 

4  causes  stop  if  no  change  in  the  Xth  (col  51  to  53)  decimal  place  of  the  composite  validity 
or  on  the  first  decrease 

5  causes  stop  after  the  first  decrease  in  composite  validity  after  X  (col  51  to  53)  items  have 
entered 

Options  2  thru  5  will  also  stop  on  fulfillment  of  Option  1. 

49  Total  number  of  subsamples  in  FRDF  (if  blank,  assumes  3) 

50  Total  number  of  subsamples  in  IKF  (if  blank,  it  is  set  equal  to  col  49) 

>1-53  Associated  with  stop  option  (col  48) 

>4-56  Subsample  sequence  for  IKF  (assumes  1,2,3) 
i9-61  Subsample  sequence  for  FRDF  (assumes  1,2,3) 

62  1  if  only  positive  unit  weighting  requested  for  composite  buildup,  blank  otherwise 

64  1  if  checkpoint  requested  every  hour  (wall  clock),  blank  otherwise 

>5-69  M  AXR  EC  M  aximum  number  of  records  to  be  written  on  the  direct  access  file  in  I  APG  6. 
The  field  may  be  blank  if  IAPC  5  and  6  are  run  together.  If  I  APG  6  is  run  separately  from 
IAPG  5,  the  value  read  here  should  be  the  number  appearing  in  the  last  message  printed 
by  IAPG  5  -  "DIRECT  ACCESS  FILE  SIZE  =XXXXX’\ 


Tide  Card(s),  required 

The  same  as  in  package  one. 

End  Title  Caid,  required 

The  same  as  in  package  one. 


Key  Change  Canls,  optional 


ID  of  item  that  is  to  have 
New  key  for  alternative  I 
New  key  for  alternative  2 
New  key  for  alternative  3 
New  key  for  alternative  4 
New  key  for  alternative  S 
New  key  for  alternative  6 


Description 


its  key  changed 
(assumes  FI 2.8 
(assumes  FI 2.8 
(assumes  FI 2.8 
(assumes  FI 2.8 
(assumes  FI 2.8 
(assumes  FI 2.8 


format) 

format) 

format) 

format) 

format) 

format) 


NOTE:  Read  stops  upon  encountering  a  blank  case  ID  field. 


i 


Item  Elimination  Cards,  optional 

Cards  contain  ID  numbers  of  items  to  be  eliminated  (right  justified  in  four-character  fields,  20 
items  per  card  for  as  many  cards  as  are  necessary).  Read  stops  upon  encountering  a  blank  field.  If  the 
number  of  items  to  be  eliminated  is  a  multiple  of  20,  there  must  be  a  blank  card  following  the  last 
item  elimination  card  to  stop  the  read. 

NOTE:  The  item  elimination  cards  for  subsample  1  are  first,  subsample  2,  second  and  subsample 

3,  third.  If  items  are  not  to  be  eliminated  from  a  subsample  and  the  item  elimination  code 
on  the  Main  Control  Card  is  1,  a  blank  card  must  be  supplied  for  that  subsample. 

NOTE:  The  input  to  IAPC  5  can  consist  of  a  maximum  of  1500  items  (original  and  dummy); 

however,  no  more  than  500  items  may  remain  after  all  program  and  user  requested 
eliminations. 
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APPENDIX  E;  DIAGNOSTIC  MESSAGES 


I.  IAPG  1  to  4  Initialization  Program  Messages.  The  following  messages  will  terminate  the 
run: 

A.  CONTROL  CARD  MISSING  -  EXECUTION  TERMINATED 

The  first  card  read  by  the  initialization  program  was  not  the  main  control  card. 

B.  NOGO  SPECIFIED.  RUN  TERMINATED. 

Control  cards,  including  data  subsample  control  cards,  are  scanned,  interpreted, 
and  checked  for  errors.  Data  are  not  processed  (in  fact  data  cards  must  not  be 
present). 

II.  I.4PC  1  Messages 

A.  The  following  messages  cause  run  termination: 

1.  ERROR  -  CONTROL  CARD  MISSING 

This  message  occurs  when  cards  of  a  data  subsample  control  card  set  are 
missing. 

2.  ERROR  -  ILLEGAL  VALUE  FOR  NUMBER  OF  RESPONSES 
FOR  ITEM  NUMBER  XXXX 

The  maximum  response  for  an  item  was  defined  as  less  than  2  or  greater 
than  5  (with  omits  valid)  or  greater  than  6  (with  omits  invalid).  The  run 
will  terminate  after  checking  the  remaining  items. 

3.  TOO  MANY  CASES  ELIMINATED,  RUN  WILL  TERMINATE  AT 
END  OF  CURRENT  SAMPLE 

4.  STOP.  EOF  FOUND  ON  DATA  UNIT 

End  of  file  (or  any  systems  card)  found  on  data  input  unit  while  still 
expecting  to  read  data. 

B.  The  following  messages  are  warnings  of  data  errors  and  will  not,  by  themselves, 
cause  data  elimination: 

1.  CRITERION  NUMBER  X  FOR  CASE  'aaaaaaaaa'  IS 
UNDEFINED. 

2.  WARNING  -  CARD  NUMBER  X  OF  CASE  aaaaaaaaa'  HAS 
THE  WRONG  CASE  ID. 

An  indicator  of  out-of-order  cards.  The  case  will  be  eliminated  if  this  is 
the  second  such  error  for  the  case. 

3.  ERROR  -  THE  RESPONSE  FOR  ITEM  XXXX  IN  CASE  ’tmaaoraor 
a  a  -  IS  X  WHICH  IS  OUTSIDE  THE  RANGE  (XX)  FOR  THIS 
ITEM.  THE  RESPONSE  HAS  BEEN  RECORDED  AS  AN  OMIT  OR 
THE  CASE  HAS  BEEN  ELIMINATED  (SEE  OMIT  VALID  CODE). 
IF  THE  LATTER  IS  TRUE,  A  STATEMENT  TO  THAT  EFFECT 
WILL  FOLLOW  IMMEDIATELY. 
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C.  The  following  messages  are  for  errors  which  will  cause  a  data  case  to  be 
eliminated  from  the  response  data  file: 

1.  ERROR  -  THE  CRITERIA  CARD  FOR  CASE  'aaaaaaaaa'  IS 
MISSING. 

2.  ERROR  -  ALL  CRITERIA  IN  CASE  'aaaaaaaaa'  ARE 
UNDEFINED. 

Elimination  does  not  occur  immediately;  the  program  will  scan  the 
response  cards  for  errors. 

3.  ERROR  -  CASE  occceca'  HAS  MORE  THAN  ONE  CARD  WITH 
THE  WRONG  ID. 

4.  ERROR  -  THE  CARD  FOLLOWING  CARD  X  IN  CASE  aaaaaaaa 
a'  IS  OUT-OF-ORDER. 

D.  After  each  message  in  IIC  above,  the  following  message  will  appear: 
**********  CASE  aaaaaaaaa'  HAS  BEEN  ELIMINATED  ********** 

E.  The  following  message  is  for  warning  purposes  only  and  is  connected  to  message 
II  A3  above: 

WARNING  -  THE  NUMBER  OF  ELIMINATED  CASES  IN  PROGRAM  1 
PLUS  THE  NUMBER  OF  CASES  THAT  WILL  BE  ELIMINATED  IN 
PROGRAM  2,  DUE  TO  INVALID  OMITS,  WILL  EXCEED  THE 
MAXIMUM  ALLOWABLE  NUMBER  OF  ELIMINATIONS  (XXXXXX)  BY 

xxxxxx. 


IAPG  2  Messages 

A.  CASE  'aaaaaaaa'  HAS  AN  INVALID  OMIT  ON  ITEM  XXXXX. 

This  message  appears  in  the  appropriate  subsample  error  roster  and  the  specified 
case  is  eliminated. 

B.  ERROR  IN  COMPUTATION  OF  T. 

Overflow  error  in  computing  the  t-value  associated  with  a  known  probability 
level  and  degrees  of  freedom  (used  in  computing  significance  keys  for  a 
continuous  criterion).  The  run  terminates. 


IAPG  3  Message 

ILLEGAL  VALUE  FOR  MAXIMUM  ALTERNATIVE 


IAPG  4  Messages 

The  following  messages  will  terminate  the  run: 


A. 

B. 


REQUESTED  KEYING  OPTION  NOT  IN  ITEM  KEY  FILE 
REQUESTED  CRITERION  NOT  IN  ITEM  KEY  FILE 


VI. 


I A PG  5  Messages 


A.  The  following  messages  will  cause  run  termination: 

1.  CONTROL  CARD  MISSING  -  EXECUTION  TERMINATED. 

The  first  card  read  by  the  program  was  not  the  I A  PC  5  to  7  main  control 
card. 

2.  *****  WRONG  KEYING  OPTION  ***** 

Keying  Option  4  (non-existent)  was  requested  on  the  main  control  card. 
Run  termination  does  not  occur  immediately  but  will  occur  when  the 
program  is  unable  to  find  the  key  on  the  item  key  file. 

;i.  NUMBER  OF  ITEMS  IN  KIRF  GREATER  THAN  500.  RUN 
TERMINATED. 

B.  The  following  message  is  a  warning  only  and  occurs  when  an  item,  which  was 
requested  by  the  user  to  be  deleted,  was  alreadv  deleted. 

ITEM  XXX  ALREADY  DELETED 


VII.  LA  PG  6  M  essages 

The  following  messages  will  cause  run  termination: 

A.  REQUESTOR  WANTS  OPTION  X.  CRITERION  X.  FILE  READ  HAS 
OPTION  X,  CRITERION  X.  W  rong  KIRF  header  read. 

B.  STOP.  ATTEMPTED  TO  READ  PAST  END  OF  KIRF  FILE  (REEL  X)  IN 
PRO. 

End-of-file  found  on  KIRF  unit  before  finding  requested  KIRF.  Possible  wrong 
reel  mounted,  etc. 

C.  STOP.  ATTEMPTED  TO  READ/WRITE  PAST  DISK  LIMITS  IN  PRO. 
Possibility  exists  that  the  direct  access  file  was  not  defined  large  enough  to 
contain  all  of  the  cases  and  correlation  records.  It  could  be  that  IAPG  6  is  being 
run  separately  from  I A  PG  5  and  that  M  A  X  R  EC  in  the  I A  PG  5  to  7  main  control 
card  was  not  defined. 


VIII.  I A  PC  7  Messages 

V.  The  following  messages  will  cause  run  termination: 

1.  STOP.  VTTF.M  PTED  TO  READ  PAST  END  OF  KIRF  FILE  (REEL 
\)  IN  M  VIN  7. 

2.  «  KONG  KIRF 

hex  or  criterion  on  KIRF  is  in  error. 

R.  The  following  messages  cause  some  processing  to  be  skipped: 

I  MINING  KIRF  \X\X 

The  required  KIRF  had  no  items:  program  will  skip  to  next  KIRF. 

2.  MISSING  ISSK  \XXX 

No  items  in  KIRF  .  set  selection  sequence  pattern  ID  to  9  and  continue  to 
next  h  I R  F. 


b.V 


•f.,- : 


I 


IX.  IAPC  5  to  7  Table  of  Contents  Message 
NO  ROSTERS,  ONLY  ONE  K1RF. 

X.  Program  SEARCH  message 

This  is  a  file  searching  subroutine  used  by  I A  PC  2  to  7.  The  message  will  cause  run 
termination. 

SEARCH  FORTRAN  UNIT  XX  DEVICE  ERROR  CODE  =  XXX.  RUN 
TERMINATED. 

This  indicates  that  the  hardware  device  for  the  above  FORTRAN  unit  was  not  a  TAPE 
drive  nor  a  FASTRAN  mass  storage  device  as  expected. 
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APPKMtIX  /•'■  Rl  N  TIM  F  EXAMPLES 


For  large  problems,  an  lAPb  eomputer  run  ean  be  very  lime  consuming.  The  wall  clock  times 
(in  hours)  for  a  problem  that  was  run  on  the  l  NIVAL  I  I  OK  at  A  Ell  Rl.  are  as  follows: 

iap<;  i  t  iap(;  5  iap<;  6-7 


55  items,  three  samples  of  500  eases  each  1/2  I  1 

55  items,  three  samples  of  100  eases  each  I/O  l/f  l/f 

The  times  given  for  I  \  l’<»  1  to  f  are  for  one  criterion  and  all  four  keving  options:  however,  the  times 
given  for  I  \  IM 5  to  7  are  for  one  criterion  and  only  one  keying  option.  No  eases  or  items  were 
eliminated  during  program  execution. 
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